Networks of Photos, Landmarks, and People

Social photo-sharing sites like Flickr contain vast amounts of latent information about the world and human behavior. We describe our recent work in building automatic algorithms that analyze large collections of imagery in order to extract some of this information. At a global scale, we show how geo-tagged photographs can be used to identify the most photographed places on Earth, as well as to infer the names and visual representations of these places. At a local scale, we show that we can build detailed 3-d models of a scene by combining information from thousands of 2-d photographs taken by different people and from different vantage points. The dramatic growth of social content sharing websites has created immense collections of user-generated visual data online. Flickr.com alone currently hosts over 4 billion images taken by more than 40 million unique users [1], while Facebook.com grows by nearly 3 billion photos every month [2]. While users of these sites are primarily motivated by a desire to share photos with family and friends, collectively they are generating vast repositories of online information about the world and its people. Each of their photos is a visual observation of what a small part of the world looked like at a particular point in time and space. It is also a record of where a particular person (the photographer) was at a moment in time and what he or she was paying attention to. In aggregate, and in combination with the non-visual metadata available on photo sharing sites (including photo timestamps, geo-tags, captions, user profiles, and social contacts), these billions of photos present a rich source of information about the state of the world and the behavior of its people. In recent work, we have shown how vast photo collections like Flickr can be used to reconstruct information about the world at both global and local scales [3,4]. At a global level, we can create annotated maps of the world completely automatically, using the geo-tags on photos to reconstruct land boundaries, using tags to infer place names, and using visual analysis to find frequentlyphotographed scenes (an example is shown in Figure 1). We can also use this analysis to generate statistics about places, such as ranking landmarks by their popularity or studying which kinds of users visit which sites. At a more local level, we can use techniques from computer vision to automatically produce strikingly accurate 3-D models of a landmark, given a large number of 2-D photos taken by many different users from many different vantage points (see Figure 2). This work is part of a larger emerging research trend within computer science that is studying how to use publicly available data from online social networking sites to address questions in a range of fields in the humanities and social sciences [5]. Compared to traditional techniques like surveys and direct measurement, data collection from online social networking sources is of negligible cost and can be conducted at unprecedented scales. The challenge is that online data is largely unstructured, thus requiring sophisticated algorithms that can organize and extract meaning from noisy data. In our case, this involves developing automated techniques that can find patterns across millions of images. In this paper, we describe our recent work in using online photo collections to reconstruct the world at both global and local scales. Fig. 1. An annotated map of North America, automatically generated by analyzing nearly 35 million photos from Flickr. For each of the top 30 most photographed cities, the map shows the name of the city inferred from tags, the name of the most photographed landmark, and a representative photo of the landmark. (© David Crandall)